Well, the usual troubleshooting steps are
start with the RetroPie image from
retropie.org.uk/download. It's a tested and mostly stock Raspbian image and the best baseline system to start experimenting.
make sure you have an adequate power supply and you don't get any under-voltage warnings.
test if the device is detected by the OS by running a few command line utilities
# see what USB devices are connected with
lsusb
# see what input devices are detected by the OS and what kind (gamepad/keyboard/mouse/etc.)
cat /proc/bus/input/devices
test if the device inputs are registered in the system by running a few joystick/gamepad testing programs fromt the command line that capture the button/joypad/d-pad presses and show them on the terminal:
# 1st joystick
jstest /dev/input/js0
# 2nd joystick
jstest /dev/input/js1
# another testing utility
evtest